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We discuss some issues arising in the evaluation of confidence intervals in the presence of nuisance parameters (system- 
atic uncertainties) by means of direct Neyman construction in multi-dimensional space. While this kind of procedure 
provides rigorous coverage, it may be affected by large overcoverage, and/or produce results with counterintuitive 
behavior with respect to the uncertainty on the nuisance parameters, or other undesirable properties. We describe 
a choice of ordering algorithm that provides results with good general properties, the correct behavior for small 
uncertainties, and limited overcoverage. 



1. Introduction 

A conceptually straightforward method to incorpo- 
rate systeniatics into Confidence Limits is to apply 
the usual Neyman construction directly on the com- 
plete pdf of the problem, including the set of addi- 
tional parameters v describing the systematic effects, 
and then project the solution on the space of parame- 
ters of interest /i. Systematic uncertainties may take 
the form of an allowed range for the v^s, or may be 
defined by the observables of the problem. Although 
the method can be applied to a more general situa- 
tion, we will assume in the following discussion that 
measurements are available of some ( "subsidiary" ) 
observable(s) y, whose only purpose is to provide in- 
formation on the systematic parameters, through the 
dependence of their pdf on v. In this case, one will 
consider the overall pdf. 



p{{x,y)\{fi,u)) 



(1) 



that gives the joint probability of observing the value 
of the "physics observables" x plus all "systematic 
measurements" y, given all unknown parameters, 
physics and systematics. One starts by deriving Con- 
fidence Limits in the larger (^, space from the ob- 
served values of {x, y) with the same procedure that 
could have been used in absence of systematics to 
derive limits on ji: one simply needs to sample a 
number of points inside the parameter space and re- 
quire coverage for each of them. Then, in order to get 
results containing only the physical parameters, one 
needs to project the confidence region in (/x, v) onto 
the ^ space, so as to get rid of unwanted information 
on the nuisance parameters. 

Although the above procedure is general, con- 



ceptually simple, and rigorous, other methods have 
been preferred in the vast majority of problems in 
physics. This can be ascribed to a few important 
difficulties with this method. To begin with, the 
problem of numerical calculation of Confidence Re- 
gions (CRs) in multi-dimensional spaces is often 
quite complex and CPU-consuming. Then there is 
a non-trivial question of what ordering algorithm to 
use in the Neyman construction. There is an issue of 
"efficiency", or power, of the solution, because pro- 
jecting the band on the /x space effectively means to 
inflate a limited region in (/x, v) to an unlimited band 
in the v direction, thereby increasing the coverage 
for all additional points (/i, v) included. This means 
that the final limits quoted on /i will almost always 
overcover, and sometimes badly, especially when ly 
has many dimensions; this is indeed the case with 
standard choices of ordering^. A related additional 
problem is that the behavior of the limits when the 
systematic uncertainty approaches zero is in many 
cases unsatisfying. It often happens that the limit 
for small systematics is quite different from the re- 
sult one would quote in absence of that systematic; 
this problem, however, is not unique to the projec- 
tion method. 

If the above problems could be alleviated, this 
methodology could find greater use in HEP. 

2. A Benchmark problem 

Our discussion, although general, will be centered on 
a specific problem that has been the initial motiva- 
tion for this work: a Poisson distributed signal in 
presence of a known background, with a systematic 
uncertainty on the signal normalization (efficiency). 
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We have: 

X Pois{efi + b) , e ~ G{e, a) (2) 

where e is the result of a subsidiary measurement 
with rcsohition cr of the unknown efficiency e, which 
is intended to be a generic "normalization factor", 
not necessarily smaller than one. In the follow- 
ing we will mostly assume a normal distribution 
for G for simplicity; the possibiUty of negative val- 
ues of the efficiency estimate does not pose any 
problems to the algorithms discussed in this docu- 
ment. This can actually occur, for instance, when 
the efficiency measurement implies some sort of 
background-subtraction procedure*^. 

3. Looking for an optimal band 

What one would like to accomplish is to find a clever 

enough rule for constructing the initial Confidence 
Band, to minimize the amount of unnecessary cover- 
age added when the band is projected onto the "in- 
teresting parameters" space. It is not obvious what 
the minimum is for a particular problem, because 
the frequentist requirement of minimum coverage for 
every possible true value of the parameters may im- 
ply some minimum amount of ovcrcovcragc, which is 
unavoidable regardless of the algorithm used in the 
construction, much in the way overcoverage occurs 
in discrete problems. Therefore, there is no reason 
for being a-priori discouraged about the capability of 
the projection method to provide powerful solutions 
(that is, narrow intervals). A striking demonstra- 
tion of this is provided by the use of the projection 
method, with an appropriately designed algorithm 
for band construction, in producing a more efficient 
solution to a classical, well-explored problem like the 
ratio of Poisson means^. 

It is intuitively obvious that in order to obtain 
an efficient solution, the initial confidence band must 
extend as far as possible along the direction of the 
nuisance parameter. This is not trivial to achieve, 
since the band needs to be built in the {x, e) space, 
while the objective is to produce a desired shape in 



^This simple and common example has been selected by the 
CDF statistics committee as a benchmark in performing com- 
parisons between a number of different methods. A minor dif- 
ference from the current example is that a positive, Poisson- 
like distribution is assumed for the subsidiary measurement in- 
stead of a Gaussian, in order to avoid problems with Bayesian 
treatment^ . 



the (/i, e) space. A good general requirement to im- 
pose is that, given any two sections of the band at 
two fixed values of the nuisance parameter e, one 
must be completely included in the other. It is in- 
tuitive that a band cannot be optimal if it does not 
satisfy this requirement, because if one had to take 
one of the two sections and expand it to completely 
include the other, the projected confidence region in 
/i would be unaffected, and conversely (me could ex- 
ploit the coverage gained in this way to trim a part 
of the exceeding part of the chosen section, thus cre- 
ating the conditions for a tightening of the projected 
confidence region. 

4. Ordering Algorithm 

One way to define how to construct the confidence 
band in the complete space is to derive it from 

an ordering fimction /(.x, e; /i, e), so that the confi- 
dence band is defined by the inequality f{x, e; /U, e) > 
c(/U, e), where the threshold c is determined for each 
value of the parameters from the usual Neyman's re- 
quirement of coverage: 

/ p{x,e\ii,e)dxde>CL (3) 

J f {x ,e;iJ,,e)>c{iJ,,e) 

where CL is the desired Confidence Level. It is 
worth noting that this is not the only conceivable way 
to define a band satisfying the coverage condition'^, 
but it is attractive for reasons of simplicity. A sim- 
ple way to implement in an ordering algorithm the 
requirement of inclusion formulated in the previous 
section is to impose that f{x,e;ii,e) is independent 
of e: f{x,e;ii,e\) = f{x,e; ^,€2)- In this way, sec- 
tions taken at different e for the same value of /U will 
only differ in the value of c(/i,e), and will therefore 
be included in one another. This requirement is also 
very convenient from the point of view of computing, 
as it implies that the ordering function / need only 
be calculated once for every /x. 

As an additional requirement, we want the pro- 
jected confidence regions to converge to the results 
in absence of systematic uncertainty when the size of 
the uncertainty goes to zero. We do not restrict to a 
specific ordering (one may want to be able to choose, 
for instance, between central and upper limits), so we 
start from a given generic ordering function fo{x; fi) 
in the restricted space. This defines the behavior of 
the ordering function along the direction of observ- 
able X, but careless extension of any such rule to the 
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whole {x,e) space will not work. As an example, 

extending the trivial ordering used to achieve upper 
limits {fo{x; fj,) = x) results in substantial overcover- 
age (see fig. la). We need additional criteria to ensure 
proper behavior in the subsidiary observable e. We 
don't want to give special preference to any values, 
because this will amount to attempting to extract in- 
formation on the nuisance parameter, while we want 
to maximize information on the physical parameter 
/Lt. We do this by choosing the following ordering 
function: 

f{x,e;ii)= p{x'\e;iJ,,e{e))dx' (4) 

J fo{x')<fo{x) 

where g(e) is the maximum Likelihood estimate 
of e for the given e. That implies that the same 
integrated conditional probability will be contained 
in the band for each value of e. 

We make an exception to the rule of being in- 
different to the value of e, e for very unlikely values: 
we select an interval of values [emin, ^max] such that 
the probability for a measurement to fall outside is 
■C 1 — CL, and assign lowest rank to all points ly- 
ing outside this interval. Prom the above conditions, 
they will never be reached by the ordering procedure, 
so they can simply be ignored, which saves computa- 
tion. This clipping technique has already been advo- 
cated as a help in keeping the projections small^; in 
our context however it seemed to have no significant 
effects beyond saving computation. 

5. Results 

We have applied the ordering rule of equation (4) 
to our problem of choice (sec. 2), with an order- 
ing /o corresponding to upper limits. Fig. lb shows 
that this time very little overcoverage is obtained, ex- 
cept from some discretization-related "ripples". It 
is interesting to note that these limits are tighter 
than the limits obtained with other popular meth- 
ods (compare, for instance, the coverage obtained 
for the same problem with Bayesian^ or Cousins- 
Highland methods^' although guaranteed by con- 
struction to cover for every possible value of both fj, 
and e. This confirms the capability of the projection 
method to produce powerful results, when used in 
conjunction with an appropriate ordering algorithm, 
as per Eq. (4). 

The procedure we have described can be used 
with any other desired ordering. If we apply it to 



Unified Intervals^, we find an interesting fact: be- 
cause of the Likelihood Ratio theorem, ensuring the 
independence of the distribution from true parame- 
ter values, the ordering algorithm defined by Eq. (4) 
is approximately equivalent to ordering based on the 
ratio of profile Likelihoods. That quantity has been 
suggested as a good intuitive ordering to use in han- 
dling systematics since ^, and has been used in neu- 
trino experiments^' ^ (with a conditional frequentist 
motivation), and in a problem very similar to ours, 
the Poisson with uncertainty on background^. It 
reappears here as an approximation of the more gen- 
eral rule defined by Eq. (4). Fig 2 shows that cover- 
age plot for our benchmark problem, which is close 
to the nominal constant 0.9, indicating that there is 
very little to be further gained. 

6. Continuity 

One of our initial goals was to obtain a continuous 
behavior when asyst 0. In previous examples, al- 
though the limit is approximated much better than 
with other frequentist methods (see for instance^*^), 
there is still a slight difference. For instance, the 
upper limit with the Unified method at 90% for 
n = 4, 6 = 3 is^ 5.6 , while our results approach 
~ 5.47 when a — > 0. More annoyingly, the limit 
found with systematics is lower. This is a well known 
problem, tied to the transition between discrete and 
continuous regime^^, and is pretty much independent 
of the specific algorithm. However, our method for 
evaluating limits allows a very simple fix, requiring 
no alterations to the ordering: all that is needed is 
to keep the size of the grid used in the numerical 
calculations from becoming too small in the direc- 
tion of the nuisance observable. This has a natural 
justification under the same principles that guided 
the general design of our algorithm: we are trying to 
disregard detailed information on the the subsidiary 
observable, in favor of information on the physics pa- 
rameter /i. In our problem, by choosing a minimum 
step Ae = 0.1 we obtain perfect continuity at zero 
(fig. 3). A side effect of this limitation is to save some 
computing time. 

7. Systematic uncertainties given as 
ranges 

The approach we have described has wider applica- 
tion than the examples mentioned above. For in- 
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stance, it can handle in a natural way the impor- 
tant situations in which no subsidiary measurement 
in available to provide information on the nuisance 
parameter. This often occurs in real life: the system- 
atic uncertainty may be due to a physical constraint, 
or related to a choice within a range (discrete or con- 
tinuous) of theoretical predictions or assumptions, or 
can otherwise be specified in a way that is not de- 
tailed enough to uniquely identify a probability dis- 
tribution. In these cases, usually the only available 
information on e is represented by a range of admis- 
sible values. 

This situation is automatically handled by our 
approach: one simply has one less observable to 
worry about, but the rest of the construction works 
exactly in the same way. In fact, calculations are 
much faster with the lack of a subsidiary measure- 
ment, so that when dealing with small systematics it 
is actually more convenient to transform any possi- 
ble nuisance measurement into an appropriate range 
for the nuisance parameter, and simply use that in- 
formation as input, in order to save computing time. 
Again, our tests yielded very limited overcoverage, 
compatible with what was required simply by the 
discrete nature of the problem. 

It is worth noting that a range of values is not 
at all equivalent to a uniform distribution, which im- 
plies more precise knowledge. For instance, by com- 
paring the limits obtained in the two cases, it is seen 
that the limits for the range case are looser then in 
the uniform distribution intuitively expected 

due to the smaller information content in a statement 
about a range (see Table 1). This is in contrast with 
what happens in a Bayesian approach, where a prior 
function is always required, and a uniform distribu- 
tion is often chosen to represent lack of information. 

In general, treating systematic uncertainties as 
ranges is a good candidate approach to problems 
with many nuisance parameters, as it allows big sav- 
ings in CPU time, in addition to avoiding the trouble 
of having to worry about the accuracy of the distri- 
butions assumed to represent the systematic uncer- 
tainties. 



8. Conclusions 

We have presented a general method to incorporate 
systematic uncertainties in a limit calculation in a 
rigorous frequentist way, which is powerful (does not 



a) 




Fig. 1. Coverage plots for upper limits with a naive ordering 
(a), and with the ordering of eq. (4) (b). The efficiency is 
measured with a Gaussian uncertainty, a = 0.1. 

produce large overcoverage), has the right limit for 
small uncertainties, can be used even with uncer- 
tainties given as ranges, and can easily be calculated 
in practice. This is based on projection of a tra- 
ditional Neyman construction with an ordering al- 
gorithm specified by Eq. (4). We have applied it to 
the specific problem of Poisson measurement with an 
uncertainty on the efficiency. 
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Table 1. Confidence Limits for Poisson+background with sys- 
tematic uncertainty on the efficiency, obtained by extending 
Unified Limits through Eq.{4). Results are given for 6 = 3, 
e = 1.0, and various models of uncertainty on e (see text). 





Without 


Gaussian 


Uniform 


Range 




systematics 


cr = 0.1 


±0.15 


±0.15 





0.00 , 1.08 


0.0 , 1.1 


0.0 , 0.9 


0.0 , 1.0 


1 


0.00 , 1.88 


0,0 , 1.9 


0.0 , 1.7 


0.0 , 1.9 


2 


0.00 , 3.04 


0.0 , 3.0 


0.0 , 2.7 


0.0 , 3.0 


3 


0.00 , 4.42 


0.0 , 4.4 


0.0 , 4.0 


0.0 , 4.5 


4 


0.00 , 5.60 


0.0 , 5.9 


0.0 , 5.4 


0.0 , 6.0 


5 


0.00 , 6.99 


0.0 , 7.4 


0.0 , 6.9 


0.0 , 7.4 


6 


0.15 , 8.47 


0.0 , 8.9 


0.2 , 8.2 


0.1 , 8.9 


7 


0.89 , 9.53 


0.9 , 10.3 


1.0 , 9.6 


0.8 , 10.4 


8 


1.51 , 10.99 


1.4 , 11.7 


1.5 , 10.9 


1.3 , 11.8 


9 


1.88 , 12.30 


2.0 , 13.1 


2.1 , 12.3 


1.9 , 13.1 




Fig. 2. Coverage plot for Unified limits, Gaussian uncer- 
tainty, 6 = 3, (T = 0.1. 
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Fig. 3. Behavior of upper limit when o" — > 0, same problem 
as in Fig. 2. 
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